3d camera system for infant monitoring

ABSTRACT

Systems and methods employing a depth-sensing camera to detect abdomen rise and fall during an infant sleep period, or lack thereof due to respiratory arrest. Visual processing of image data collected by one or more 3D digital camera is performed to detect the infant and to measure a distance between an infant&#39;s abdominal cavity and a camera baseline over time. An alarm may be triggered at by the system, locally and/or at remote devices, such as a mobile phone, tablet, laptop, etc. The monitoring system may also detect situations when an infant rolls from a back-sleeping to a belly-sleeping position.

BACKGROUND

Sudden infant death syndrome (SIDS), also known as crib death, is thesudden unexplained death of a child less than one year of age. SIDSusually occurs during sleep and is currently believed to be associatedwith respiratory issues. Typically, infant death occurs between thehours of 00:00 and 09:00 with no noise detected. SIDS was the thirdleading cause of death in children less than one year old in the UnitedStates in 2011. About 90% of cases happen before six months of age, withit being most frequent between two months and four months of age.

Conventional approaches to detecting an infant respiratory problem inreal time require physical attachment of devices to the infant, or havedifficult coping with background noise within the infant's environment.A more robust system for infant respiratory monitoring is thereforeneeded.

Digital cameras are often included in infant monitoring systems, forexample installed in a nursery. Such systems typically relay video data,or audio/video (A/V) data, from the camera to a remote display screen.Allowing a caregiver to visualize the infant, for example during adaytime napping period. Such systems are however relativelyunsophisticated, lacking significant image analysis capability.

BRIEF DESCRIPTION OF THE DRAWINGS

The material described herein is illustrated by way of example and notby way of limitation in the accompanying figures. For simplicity andclarity of illustration, elements illustrated in the figures are notnecessarily drawn to scale. For example, the dimensions of some elementsmay be exaggerated relative to other elements for clarity. Further,where considered appropriate, reference labels have been repeated amongthe figures to indicate corresponding or analogous elements. In thefigures:

FIG. 1 is a schematic of a 3D camera system for infant monitoring, inaccordance with some embodiments;

FIG. 2 is a flow diagram of a method for infant monitoring with a 3Dcamera system, in accordance with some embodiments;

FIG. 3 is a schematic of infant abdominal cavity distance measurementwith a 3D camera system, in accordance with some embodiments;

FIG. 4 is a schematic of infant roll detection with a 3D camera system,in accordance with some embodiments;

FIG. 5A is a plot of infant abdominal cavity distance over time, inaccordance with some embodiments;

FIG. 5B is a schematic illustrating a region of interest (ROI) overwhich depth data is filtered and analyzed for changes with time, inaccordance with some embodiments;

FIG. 6 is a flow diagram of a method for infant monitoring with a 3Dcamera system, in accordance with some embodiments;

FIG. 7 illustrates a 3D camera platform for infant monitoring, inaccordance with some embodiments;

FIG. 8 is a block diagram of a data processing system, according to someembodiments; and

FIG. 9 is a diagram of an exemplary ultra-low power system with a 3Dcamera-based respiratory monitor, in accordance with some embodiments.

DETAILED DESCRIPTION OF EXEMPLARY EMBODIMENTS

One or more embodiments are described with reference to the enclosedfigures. While specific configurations and arrangements are depicted anddiscussed in detail, it should be understood that this is done forillustrative purposes only. Persons skilled in the relevant art willrecognize that other configurations and arrangements are possiblewithout departing from the spirit and scope of the description. It willbe apparent to those skilled in the relevant art that techniques and/orarrangements described herein may be employed in a variety of othersystems and applications beyond what is described in detail herein.

Reference is made in the following detailed description to theaccompanying drawings, which form a part hereof and illustrate exemplaryembodiments. Further, it is to be understood that other embodiments maybe utilized and structural and/or logical changes may be made withoutdeparting from the scope of claimed subject matter. Therefore, thefollowing detailed description is not to be taken in a limiting senseand the scope of claimed subject matter is defined solely by theappended claims and their equivalents.

In the following description, numerous details are set forth, however,it will be apparent to one skilled in the art, that embodiments may bepracticed without these specific details. Well-known methods and devicesare shown in block diagram form, rather than in detail, to avoidobscuring more significant aspects. References throughout thisspecification to “an embodiment” or “one embodiment” mean that aparticular feature, structure, function, or characteristic described inconnection with the embodiment is included in at least one embodiment.Thus, the appearances of the phrase “in an embodiment” or “in oneembodiment” in various places throughout this specification are notnecessarily referring to the same embodiment. Furthermore, theparticular features, structures, functions, or characteristics describedin the context of an embodiment may be combined in any suitable mannerin one or more embodiments. For example, a first embodiment may becombined with a second embodiment anywhere the particular features,structures, functions, or characteristics associated with the twoembodiments are not mutually exclusive.

As used in the description of the exemplary embodiments and in theappended claims, the singular forms “a”, “an” and “the” are intended toinclude the plural forms as well, unless the context clearly indicatesotherwise. It will also be understood that the term “and/or” as usedherein refers to and encompasses any and all possible combinations ofone or more of the associated listed items.

As used throughout the description, and in the claims, a list of itemsjoined by the term “at least one of” or “one or more of” can mean anycombination of the listed terms. For example, the phrase “at least oneof A, B or C” can mean A; B; C; A and B; A and C; B and C; or A, B andC.

The terms “coupled” and “connected,” along with their derivatives, maybe used herein to describe functional or structural relationshipsbetween components. It should be understood that these terms are notintended as synonyms for each other. Rather, in particular embodiments,“connected” may be used to indicate that two or more elements are indirect physical, optical, or electrical contact with each other.“Coupled” may be used to indicated that two or more elements are ineither direct or indirect (with other intervening elements between them)physical, optical, or electrical contact with each other, and/or thatthe two or more elements co-operate or interact with each other (e.g.,as in a cause an effect relationship).

Some portions of the detailed descriptions provide herein are presentedin terms of algorithms and symbolic representations of operations ondata bits within a computer memory. Unless specifically statedotherwise, as apparent from the following discussion, it is appreciatedthat throughout the description, discussions utilizing terms such as“calculating,” “computing,” “determining” “estimating” “storing”“collecting” “displaying,” “receiving,” “consolidating,” “generating,”“updating,” or the like, refer to the action and processes of a computersystem, or similar electronic computing device, that manipulates andtransforms data represented as physical (electronic) quantities withinthe computer system's circuitry including registers and memories intoother data similarly represented as physical quantities within thecomputer system memories or registers or other such information storage,transmission or display devices.

While the following description sets forth embodiments that may bemanifested in architectures, such system-on-a-chip (SoC) architecturesfor example, implementation of the techniques and/or arrangementsdescribed herein are not restricted to particular architectures and/orcomputing systems, and may be implemented by any architecture and/orcomputing system for similar purposes. Various architectures employing,for example, multiple integrated circuit (IC) chips and/or packages,and/or various computing devices and/or consumer electronic (CE) devicessuch as set-top boxes, smartphones, etc., may implement the techniquesand/or arrangements described herein. Further, while the followingdescription may set forth numerous specific details such as logicimplementations, types and interrelationships of system components,logic partitioning/integration choices, etc., claimed subject matter maybe practiced without such specific details. Furthermore, some materialsuch as, for example, control structures and full software instructionsequences, may not be shown in detail in order not to obscure thematerial disclosed herein.

Certain portions of the material disclosed herein may be implemented inhardware, for example as logic circuitry in an image processor. Certainother portions may be implemented in hardware, firmware, software, orany combination thereof. At least some of the material disclosed hereinmay also be implemented as instructions stored on a machine-readablemedium, which may be read and executed by one or more processors(graphics processors and/or central processors). A machine-readablemedium may include any medium and/or mechanism for storing ortransmitting information in a form readable by a machine (e.g., acomputing device). For example, a machine-readable medium may includeread only memory (ROM); random access memory (RAM); magnetic diskstorage media; optical storage media; flash memory devices; electrical,optical, acoustical, or other similarly non-transitory, tangible media.

One or more system, apparatus, method, and computer readable media isdescribed below for respiratory monitoring based on collection andprocessing of 3D image data. In some embodiments, abdominal cavitymovement is assessed based on depth maps determined for each of aplurality of image frames collected over a time interval.

In some embodiments, an infant sleep period is monitored by arespiratory monitor mounted over a crib with a field of view of a 3Dcamera including the entire crib mattress. In some embodiments, objectdetection algorithms, such as, but not limited to, facial recognitionalgorithms, are employed to detect the infant. In further embodiments, afailure in facial recognition triggers an alert that the infant may haverolled to a belly-sleeping position, or the infant's face is otherwiseoccluded. A region of interest (ROI) within the image frame may beselected, for example an abdominal cavity region determined based on theposition of a detected object. In some embodiments, a depth map isdetermined for each of a plurality of image frames within at least theROI. The depth map may be stored in associated with a timestamp anddepth values analyzed as a function of time to determine abdominalcavity movement as a technique for monitoring respiration.

In some embodiments, a respiratory cycle is determined based on thedepth values stored as a function of time. In further embodiments, therespiratory cycle is assessed relative to one or more respiratory cyclemetrics. In some exemplary embodiments, an alert is triggered if eitherfrequency or depth (i.e., magnitude) of the respiratory cycle fails tosatisfy one or more predetermined thresholds.

FIG. 1 is a schematic of a 3D camera-based infant monitoring system 101,in accordance with some embodiments. System 101 includes a 3D cameraplatform 120, which is configured to output at least one of 3D imagedata, collected respiratory data (e.g., a time log of respiration), orone or more alert. 3D camera platform 120 is positioned within system101, ideally so that a geometry of the world scene is known with respectto the camera. In the exemplary embodiment, 3D camera platform 120 isintegrated into a nursery fixture, such as a mobile 115. Mobile 115includes a mount 116, for example to attached to a room ceiling overcrib 110, or to crib 110, and one or more arms 117 from which sensoryobjects 118 are suspended. In this exemplary embodiment, 3D cameraplatform 120 is mounted to have image sensors collecting image data of aworld scene encompassing the entire interior of crib 110. 3D cameraplatform 120 is positioned above the infant subject. During operation,system 101 collects image data for a camera field of view includinginfant subject 105. While mobile integration may provide an advantageousviewpoint where the depth dimension is perpendicular to the planarsurface of a mattress in crib 110. 3D camera platform 120 may bealternatively mounted directly to crib 110, or other furniture orceiling fixture (not depicted). 3D camera platform 120 may include oneor more camera module mounted in any manner known to be suitable for ahome security camera. For example, single camera platforms may bemounted in two or more corners of a room in which crib 110 is disposed.

3D camera platform 120 includes one or more digital cameras operable tocollect information from which depth (i.e., range) information may bedetermined for an image frame. Any cameras known to be capable ofgenerating 3D image frame data may be employed. In some embodiments, 3Dcamera platform 120 includes at least one of: a stereo camera or otherarray camera, time-of-flight (TOF) camera, or structured light camerasystem.

In some stereo camera embodiments, 3D camera platform 120 is a mobilecomputing device including a plurality of camera hardware modules (CM)with a predetermined baseline relationship. In some exemplaryembodiments, three camera hardware modules are included. However, anynumber of camera hardware modules and/or image sensors may be includedin an array camera. Each of the plurality of camera modules associatedwith 3D camera platform 120 may collect an input image captured from adifferent camera viewpoint. In exemplary embodiments, images capturedfrom the different viewpoints are captured at substantially the sameinstant of time such that they contain image data for a given scene. Oneof the three image frames may be designated as a reference and combinedinto an image frame having depth or range (z) information, or disparityinformation indicative of depth for each 2D (x,y) spatial positionwithin the image frame. For example, where one of the CM has a higherresolution (e.g., 8 megapixel, or more) than the other camera modules(e.g., 720p, HD, etc.), the high resolution CM may provide a defaultreference RGB/IR image. The lower resolution cameras may be consideredsupplemental to the reference and are each associated with predeterminedbaseline vector (length and direction) from the higher resolution CM togenerate depth data (D). The 3D image output from 3D camera platform 120may then be associated with RGB/IR-D data for each 2D spatial positionwith the input image frame.

In an exemplary embodiment where 3D camera platform 120 is integratedinto mobile 115, the baseline vector between the reference camera moduleand each supplemental camera module may have a length of tens ofmillimeters to tens of centimeters, depending on the form factor. In oneexemplary embodiment 3D camera platform 120 includes two lowerresolution CM modules disposed on opposite sides of one higherresolution CM spaced by known distances from the higher resolution CM.Distances between 3D camera platform 120 and a mattress within crib 110may be 1-3 meters for example. In other embodiments, where 3D cameraplatform 120 employs more than one separate infrastructure camerafixtures, baseline lengths may be on the order of meters.

In some embodiments, 3D camera platform 120 performs image processingaccording to one or more respiratory monitoring algorithm and streamsout a respiratory log in substantially real time to one or more remotelocation during operation of system 101. In some embodiments, 3D cameraplatform 120 streams 3D image data in substantially real time to one ormore remote location. For such embodiments, 3D camera platform 120 mayalso perform image processing according to one or more respiratorymonitoring algorithm in substantially real time, or not. In someembodiments, 3D camera platform 120 triggers an auditory and/or visiblealert (e.g., siren, alarm, etc.) according to one or more respiratory orother monitoring algorithm. The alert may be local to crib 110, ortransmitted to a remote device through a wired and/or wireless network.In some embodiments, one or more of 3D image data, logged respiratorydata, or alerts is communicated by 3D camera platform 120 over a localarea network 151 to a mobile computing platform 160 (e.g., smart phone,tablet computer, etc.) associated with a caregiver, or system user 165.In some embodiments, one or more of 3D image data, logged respiratorydata, or alerts is communicated by 3D camera platform 120 over a widearea network 152 to mobile computing platform 160 (e.g., smart phone,tablet computer, etc.) associated with system user 165. In someembodiments, one or more of 3D image data, logged respiratory data, oralerts is communicated by 3D camera platform 120 over wide area network152 to a system user 165 and/or an emergency responder 170, such as anemergency medical technician.

3D camera platform 120 may encode image frames where a wireless or wireddata channel has insufficient bandwidth to timely send the frame data inan uncompressed format. Depending on the available channel bit rate, agiven frame may be compressed to provide a higher or lower qualityrepresentation. In some embodiments, 3D camera platform 120 complieswith one or more wireless display specifications (e.g., WiDi v3.5 byIntel Corporation, and Wi-Fi Display v1.0 or WFD from the Miracastprogram of the Wi-Fi Alliance) have been developed for the transmissionof compressed graphics/video data and audio data streams over wirelesslocal area networks. For example, current wireless display technologiesutilizing WiFi technology (e.g., 2.4 GHz and 5 GHz radio bands) arecapable of streaming encoded full HD video data as well as high fidelityaudio data (e.g., 5.1 surround).

FIG. 2 is a flow diagram of a method 201 for infant monitoring with a 3Dcamera system, in accordance with some embodiments. In some embodiments,method 201 is performed by monitoring system 101 (FIG. 1). In exemplaryembodiments, method 201 is performed by computer processor(s) integratedinto 3D camera platform 120. In other embodiments, at least a portion ofmethod 201 is performed by a computer processor remote from 3D cameraplatform 120.

Method 201 begins with system calibration at operation 205. Calibrationoperation 205 may be divided between environment-specific calibrationand subject-specific calibration. Calibration operation 205 may also bedependent on the 3D camera platform employed to implemented method 201.TOF, structured light, and stereo camera platforms each have knowncalibration techniques for determining world scene geometry, and any ofthem may be employed at operation 205 to determine the geometry of thefield of view with respect to the camera baseline. Initialauto-calibration routines may be performed to arrive at accuratelymeasured real-world range (i.e., distance, or depth) between the 3Dcamera platform baseline and a reference surface, such as that of a cribmattress.

Following the environmental calibration, accurate depth data isgenerated by platform 120. Subject-specific calibration routines maythen be performed at operation 205 for one or more of: teaching objectdetection algorithms (e.g., facial recognition), and building arespiratory cycle baseline from which one or more respiratory metricsmay be determined specific to the infant subject to be monitored. Insome embodiments, operation 205 is conducted during a controlled or“known good” sleeping period for the infant subject. In some exemplaryembodiments, operation 205 further entails performing operations 210,215, 220, and 230 (described further below) during the baseline sleepingperiod. Performance of these operations may then be distinguishedbetween the calibration phase, during which the system is generatingbaseline respiratory data, and a runtime, during which the system willissue event-based alerts at operation 290. Respiratory metricsdetermined during the calibration phase may be employed as the basis fortriggering alerts during the runtime.

At operation 210, the scene is illuminated. While scene illumination maybe within the visible (RGB) band, exemplary embodiments illuminate thescene over the near infra red (NIR) band. Such embodimentsadvantageously enable low light 3D image data collection, for exampleduring nighttime infant sleeping periods. In some embodiments, sceneillumination at operation 210 is limited to the NIR. As used herein, theRGB band extends between 400 nm up to 700 nm, while the NIR band extendsfrom about 701 nm to at least 1200 nm. For NIR illumination embodiments,at least one camera platform includes one or more NIR light emittingdiode (LED) and an RGBNIR camera sensor, which may be any known CMOSsensor without a hot mirror in the optical path, or an enhanced NIRsensor including a specific NIR sensitive pixels.

Method 201 continues at operation 215, where scene information iscollected in the form of consecutive image data frames (i.e., videoframes). In exemplary embodiments, the scene information collected atoperation 215 is through a lens having a field of view encompassing anentire area of a crib mattress within a crib at the working distance(e.g., 1-3 m). In exemplary embodiments, the scene information collectedat operation 215 includes an infant subject. The image data framescollected at operation 215 comprises 3D image data including depthinformation, in addition to color information (e.g., intensity for eachof a plurality of color channels in any color space, such as RGB,YP_(B)P_(R), YC_(B)C_(R), or the like). In some embodiments, depthinformation collected at operation 215 is in the form of a depth mapcorrelated with a plurality of pixels p_(i), each having an imagecoordinate x_(i),y_(i) associated with the input image frame. In otherembodiments, the depth information collected at operation 215 is in theform of a disparity map correlated with the plurality of pixels p_(i),each having an image coordinate x_(i),y_(i) associated with the inputimage. A disparity value associated with a pixel indicates thecorrespondence of the pixel in one image (e.g., collected by a firstcamera module in the 3D camera platform) to a pixel in another image(e.g., collected by a second camera module in the 3D camera platform).The disparity estimation may be by any technique, as embodiments hereinare not limited in this respect. There may be a depth or disparity valueassociated with each pixel of an image frame, or some depth/disparityvalues may be absent in the event of a camera occlusion and/ortextureless surface, etc. For some embodiments, a 3D (X,Y,Z) spatialcoordinate map is generated at operation 215. Any mapping functions maybe utilized to determine 3D spatial coordinates from a disparity valueat the corresponding pixel position, for example based on predeterminedcamera parameters.

In the exemplary embodiment illustrated in FIG. 2, an object within theimage data frames is detected and tracked at operation 220. A real-timevisual object detection and/or tracking algorithm may be employed todetect and track the infant subject across consecutive frames. Oneobjective of tracking is to associate objects in consecutive images,based on the detection or tracking of previous image frames. Real-timevisual object tracking may entail processing the video data stream atthe camera frame-rate to determine automatically a bounding box of theinfant subject, or determine that the object is not visible, in eachframe. Infant movement is expected to be limited during a sleepingperiod, and so many challenges associated with real time object trackingare reduced in the exemplary embodiments. Adaptive tracking-by-detectionmethods are widely used in computer vision for tracking arbitraryobjects, and any such technique may be employed at operation 220 todetermine the location of the infant object in relation to the 3D cameraplatform.

In some embodiments, the object detected includes one or more facialfeature. Embodiments may employ any known facial feature detectionmethods, such as, but not limited to, geometry and color-basedtechniques. Although eyes are often employed, for exemplary embodimentswhere monitoring during sleep periods is desired, facial recognitionfeatures exclude the eyes (e.g., are limited to nose, mouth, head).Other techniques, including blob detection, may be employed to detectand track infant shape, which is a relatively predictable form and isalso expected to be relatively static over a sequence of images capturedfrom a camera spanning an infant sleeping period. Thermal imaging datamay also be collected by the 3D camera platform, and employed fortracking position of the subject infant.

At operation 230, a relative position of an abdominal cavity of theinfant subject is tracked over time. In other words, abdominal cavitymovement relative to the 3D camera platform in a fixed position istracked). In exemplary embodiments, abdominal cavity movement is trackedon the basis of depth information collected from a 3D camera platform.FIG. 3 is a schematic of infant abdominal cavity distance measurementwith a 3D camera system, in accordance with some embodiments. As shown,over three points in time t₁, t₂, t₃ representing a full respiratorycycle, a distance between a baseline associated with 3D camera platform120 and a surface associated with the abdominal cavity of infant 105varies from z₁ at peak exhale, to z₂ at peak inhale, and back to z₁.

Noting that infants are predominantly belly breathers, the abdominalcavity of an infant may be expected to move by 3 mm, or more, during therespiratory cycle including exhalation and inhalation half cycles.Noting further that it is generally desirable for infants to sleep ontheir backs, for example to avoid potential respiratory problems,movement of the abdominal cavity is then primarily in a directionperpendicular to a mattress surface during the sleeping period. Movementof the abdominal cavity is then primarily in the depth dimension of a 3Dcamera platform advantageously positioned directly above the cribmattress as illustrated in FIG. 3. For 3D camera platforms positioned inlocations other than directly above the crib mattress, greater depthresolution may be needed as a function of geometry. 3D camera platformswith at least 3 mm depth resolution are now commercially available(e.g., RealSense™ 3D camera from Intel Corporation).

In some embodiments, abdominal cavity tracking at operation 230 isconditioned upon satisfaction of a back sleeping criteria. For example,abdominal cavity tracking operation 230 may be predicated uponsuccessful object detection at operation 220. In some embodimentstherefore, an infant roll alert may be issued at operation 290 inresponse to an object detection failure at operation 220. FIG. 4 is aschematic illustrating infant roll detection with a 3D camera system, inaccordance with some embodiments. As shown, at a first point in time t₄,3D camera platform 120 successfully detects infant facial features 406for one or more image frames, and in response, depth informationassociated with these frames is processed during abdominal cavitytracking operation 230. At a later point in time t₅ (FIG. 4), 3D cameraplatform 120 fails to detect a facial features 406 for one or more imageframes, and a rollover/facial occlusion alert is issued at operation290.

In exemplary embodiments, tracking of abdominal cavity movement atoperation 230 entails analysis of depth data collected for each of aplurality of image frames collected over a monitoring time interval,which may be many hours. Image frames may be collected at some framerate associated with the 3D camera platform, for example 30, 60, fps,etc. An integrated electronic memory/storage may store compressed imagedata (e.g., 12-24 hours) with a circular buffer implemented to replacethe oldest image data with newest upon reaching memory/storage capacity.All, or a subsampling, of the image frames collected may be processed insubstantially real time to determine changes in abdominal cavityposition (i.e., changes in depth value) between frames. The abdominalcavity depth/position changes are then tracked over time to computecavity movement. Cavity position as function of time and/or cavitymovement is then compared against one or more predetermined abdominalcavity movement specifications or metrics. If the cavity movement failsto satisfy at least one monitored metric, method 201 proceeds tooperation 290 where an alert is issued upon detecting a failure. In someembodiments, failure criteria employed at operation 290 may includefacial detection, and/or respiratory rate, and/or respiratory depth ortidal force.

In some embodiments, respiratory amplitude is determined based onchanges in the distance between the camera baseline and the abdominalcavity surface over time. FIG. 5A is a plot of infant abdominal cavitydistance (i.e., depth from 3D camera platform) over time. In the plotillustrated, the solid line 501 represents continuous respiratory datarepresentation that might be derived from depth data sampled atoperation 230 (FIG. 2) with a 3D camera having some finite depthresolution and finite depth sampling frame rate. Over a first timeinterval 505, abdominal cavity distance varies from a maximum distanceto a camera baseline at a full exhale state to a minimum distance at afull inhale state (for a camera platform disposed over the infantsubject), and back to a maximum distance. This respiratory cycle occursover respiration period T₁ (i.e., respiration rate) and a respirationdepth or amplitude A₁.

Any known signal processing techniques may be applied to the depth datacollected over time to determine rates of change and/or amplitudes ofchange in depth (abdominal cavity distance) that may then be comparedagainst predetermine thresholds. For example, a difference between amaximum and minimum distances (depths) to the camera baseline may bedetermined as an indication of respiration depth/tidal volume. Inanother example, respiratory rate or period is determined based on adifference between image frame timestamps associated with consecutivepositions within the respiratory cycle. For example, in the respiratorycycle, there is a static state associated with each of the full exhale(corresponding to the maximum distance to the camera baseline) and withthe full inhale (corresponding to the minimum distance to the camerabaseline). In the illustrated embodiment, T₁ (or the reciprocalrespiration rate) is determined by computing a difference between imageframe timestamps associated with two consecutive static states where thecavity distance derivative as a function of time is zero.

FIG. 5A further illustrates variation in abdominal cavity displacementcycling over time. If the respiration period T₁ is less than apredetermined maximum time threshold (or minimum rate threshold), andrespiration amplitude A₁ is greater than a predetermined minimum tidalvolume threshold, no alert would be issued during time interval 505.During time interval 510, respiration amplitude A₂ falls below theminimum tidal volume threshold, and a first alert indicative of shallowbreathing is issued by the monitoring system. During time interval 515,at least respiration period T₂ falls below the maximum period/minimumrate threshold, and a second alert indicative of apnea is issued by themonitoring system.

In some embodiments, depth values employed in the tracking of abdominalcavity movement in the manner described above are first filteredspatially, temporally, or both. The filtered depth values are thenstored to an electronic memory with a timestamp, which may be associatedwith one or more image frame(s). In some embodiments, determiningchanges in a depth associated with an abdominal cavity further comprisesspatial filtering of depth values. One form of spatial filtering isthrough masking down a frame-wide depth map to those depth valuesassociated with pixel positions in a region of interest (ROI) within theimage frames. In some embodiments, ROI is selected based on a positionof a tracked object within the frame, such as one or more facialfeatures.

FIG. 5B is a schematic illustrating a ROI 550 over which depth data isanalyzed for changes with time, in accordance with some embodiments. ROI550 corresponds to a subset of pixels p_(i) having positions x_(i),y_(i)within a bounding box or window. In some embodiments, ROI 550 ispositioned within a frame relative to a location determined for detectedinfant facial features 406. Once an ROI is selected (or even if no ROIhas been selected) depth values may be filtered spatially using anyknown spatial filtering algorithm in an effort to extract real changesin depth values from measurement noise, for example caused by error inthe depth measurement. As one example, a Gaussian filter or otheraveraging technique may be applied within each image data frame. In someembodiments, a polynomial surface represented by the grid illustrated inFIG. 5B is fit to the 3D spatial coordinates associated with each imagedata frame. Any multivariate fitting algorithm, such as but not limitedto principal component analysis (PCA) may be employed. Depending on thelateral dimensions of the ROI, the polynomial surface fit to the depthdata may include higher order curvature terms, or not. For embodimentswhere the ROI is small, the surface may be assumed linear, and a depthvalue for a frame modeled as a hyperplane.

In some embodiments, depth values are temporally filtered over aplurality of frames. Any known temporal filtering algorithm may beapplied to extract real change in depth values from measurement noise.In some exemplary embodiments, depth values are averaged for a givenpixel position x_(i),y_(i) over a plurality of consecutive frames.Noting that the respiration rate of an infant is typically 20-40/minute.Change in depth values associated with abdominal cavity movement may befiltered over window of at least 5 consecutive frames, and as many as 10consecutive frames without losing significant depth resolution.

In some embodiments, both spatial filtering and temporal filtering areapplied to depth values collected over a time interval. In someexamples, a set of depth values temporally filtered to arrive at firstsubset of depth values, which are then spatially filtered to arrive at asecond subset of depth values, for example associated with a ROI, whichare then analyzed to determined abdominal cavity movement. In some otherexamples, a set of depth values are spatially filtered to arrive atfirst subset of depth values, for example associated with a ROI, whichare then temporally filtered to arrive at a second subset of depthvalues, which are then analyzed to determined abdominal cavity movement.

FIG. 6 is a flow diagram of a method 601 for infant monitoring with a 3Dcamera system, in accordance with some embodiments. Method 601 is oneexemplary embodiment of method 201 (FIG. 2), and further illustratesrespiratory fault detection. In some embodiments, method 201 isperformed by monitoring system 101 (FIG. 1). In exemplary embodiments,method 601 is performed by computer processor(s) integrated into 3Dcamera platform 120. In other embodiments, at least a portion of method601 is performed by a computer processor remote from 3D camera platform120.

Method 601 begins with a system calibration loop performed until thesystem is initialized. System parameters are set, reset, orbootstrapped, at operation 205. Calibration parameters are updated atoperation 205, and the calibration routine continued until calibrationcriteria are met. In some embodiments, the system is deemed calibratedupon both successful object detection and successful detection of aperiodic change in depth values satisfying predetermined respiratoryrate and depth thresholds.

Following the calibration phase, method 601 continues to operation 210,where the scene is illuminated, for example with an IR illuminator. Atoperation 215, image data frames are collected with 3D camera.Operations 210 and 215 continue at a camera frame rate throughout theduration of a runtime phase of method 601. In the exemplary embodiment,image data frames collected at operation 215 are processed through oneor more face/object detection algorithms at operation 220. In someembodiments, facial feature detection is performed for a frame as acondition to determining one or more depth values indicative of theabdominal cavity position at operation 230. For example, facial featureposition tracking between a current frame and one or more prior framemay be performed at operation 220. In some embodiments, abdominal cavityposition is then assessed at operation 230 based on depth values withinan ROI in the current frame that has been updated from that of a priorframe based on the facial feature position tracking.

In response to a feature/face detection failure, a flag may be set, andthe system waits for successful feature recognition. A time valuemaintained by first timer is assessed against a time threshold T1, whichis a predetermined maximum time for feature recognition to be lost (e.g.30 seconds). Until the first timer value exceeds time threshold T1,method 601 returns to face/objection detection operation 220 to analyzea subsequent frame. In response to he first timer exceeding timethreshold T1, an alert is issued at operation 681. In some embodiments,the alert issued at operation 681 is indicative of infant facialocclusion, for example due to an infant rolling over.

In response to successful facial feature detection, either (or both) ofcavity movement depth and rate may be quantified and compared tothresholds. Respiratory cycle depth and rate may parallel criteriaassessed independently, with facial recognition dictating whether cavitymovement assessment are made. Each of cavity movement depth and cavitymovement rate may be assessed using a two-stage “trigger-and-hold”algorithm in which a time to satisfy a predetermined criteria ismonitored and an alert issued in response to the criteria not being metwithin the allotted time. A failure to detect a threshold breathingdepth or rate therefore does not induce an alert until the failure stateholds for more than a threshold period of time. As infant respiration ismay be erratic in the short term, hold time thresholds mayadvantageously reduce a risk of false alerts. The benefit of minimizingfalse positives may be balanced with the benefit of rapid detection of arespiratory issue. System sensitivity may be adjusted through themodification of the hold criteria as well as the trigger criteria.

As further illustrated in FIG. 6, in response to successful facialfeature detection for the current frame, method 601 proceeds tooperation 645 where the first timer time value is reset. At operation230, abdominal cavity position is tracked, for example through a z-depthmeasurement as described elsewhere herein. In advantageous embodiments,facial detection operation 220 and abdominal cavity position trackingoperation 230 are substantially real time, keeping pace with image datacollection operation 215 performed at a camera frame rate. At operation230, a difference between the abdominal cavity depth (position) for thecurrent frame relative to a reference depth (position) associated withone or more prior frames is determined. In some embodiments, the changein cavity position between the current frame and reference is comparedto a minimum cavity movement threshold. In some embodiments, themovement threshold is a portion of a distance between full exhale andfull inhale determined during calibration (e.g., 3 mm). A maximum cavitymovement threshold may be implemented as well.

In response to failing to meet the cavity movement threshold, a timevalue maintained by second timer is assessed against a time thresholdT2, which is a predetermined maximum time before a next breath must bedetected (e.g. 30 seconds). In response to the second timer exceedingtime threshold T2, an alert is issued at operation 682. In someembodiments, the alert issued at operation 682 is indicative ofrespiration depth falling below the threshold.

In response to the meeting the cavity movement threshold, one breath isdeemed detected. In some embodiments, in response to the meeting thecavity movement threshold, the current frame is designated the nextreference frame from which a threshold cavity movement is to bedetected. In this manner method 601 may proceed by detecting respiratoryhalf cycles; repeatedly thresholding the incremental distance betweenexhale and inhale, and then the distance between inhale and exhale.Method 601 proceeds to operation 655 where the second timer time valueis reset. Until the second timer value exceeds time threshold T2, method601 continues to test the cavity movement rate against a predeterminedrespiratory rate threshold. In some embodiments, a rate determination isbased on a time moving average determined at operation 660. The timemoving average may be determined for a plurality of most recent framescorresponding to a time window of 30 seconds. To compute a cavitymovement rate, a number of breaths detected over the time moving timewindow is determined, for example from image frame timestamps, orcounted by a timer. The computed rate is then compared against apredetermined rate criteria. The rate criteria may be a band with bothupper and lower rate limits, or may be one-side with only and upperlimit.

In response to failing the respiration rate criteria over the timewindow ending with the current frame, a time value maintained by thirdtimer is assessed against a time threshold T3, which is a predeterminedmaximum time before the rate criteria is met (e.g., 30 seconds). Inresponse to the third timer exceeding time threshold T3, an alert isissued at operation 683. In some embodiments, the alert issued atoperation 683 is indicative of respiration rate falling below thethreshold. If the third timer has not exceeded time threshold T3, method601 returns to operation 220 where a next frame is processed. As oneexample, if a 30 sec moving average of respiration rate dropped below 25(or went over 60), a countdown period maintained by the third timerwould be triggered. Then, if the moving average rate window continued tobe out of spec for the time threshold (e.g., 30 sec), the alertthreshold is met and an alert issued at operation 683.

In response to satisfying the rate criteria as of the current frame,method 601 proceeds to operation 665 where the third timer time value isreset and method 601 returns to operation 220 where a next frame isprocessed.

FIG. 7 further illustrates 3D camera platform 120 including, inaccordance with some embodiments. FIG. 7 further illustrates howrespiratory monitoring components may be integrated with various othercomponents of the 3D camera platform 120. Platform 120 may be aninfrastructure device with a grid-based power supply (i.e., plugged-in),or a mobile computing device. A mobile computing device may refer to anydevice having a processing system and a mobile power source or supply,such as one or more batteries, for example. Examples of a mobilecomputing device may include a laptop computer, tablet, touch pad,portable computer, handheld computer, palmtop computer, personal digitalassistant (PDA), cellular telephone, combination cellular telephone/PDA,television, smart device (e.g., smartphone, tablet or smart television),mobile internet device (MID), messaging device, data communicationdevice, and so forth.

3D camera platform 120 includes hardware CM 710, 711, and 712. In theexemplary embodiment, CM 710 further includes a RGB(NIR) camera sensor758 while CM 711 and 712 each include a RGB(NIR) camera sensor 759.Sensor 758 may be a HD, FHD, QXGA, WQXGA, QSXGA, or UHD format digitalimage device, for example. In some embodiments, sensor 758 has at least8-megapixel resolution. Sensors 759 may be a HD, FHD, QXGA, WQXGA,QSXGA, or UHD format digital image device, for example. In someembodiments, sensors 759 have a lower pixel resolution than sensor 758,for example 1-5 mega pixel. 3D camera platform 120 may thereforegenerate three image frames concurrently, for example to provide RGBand/or IR image data and image depth data for an input image. Inexemplary video embodiments, sensors 758, 759 output multipleconsecutively exposed frames of a scene illuminated by an on-boardillumination source, such as NIR LED 798 and/or RGB LED 799.

Camera sensors 758, 759 may provide a color resolution of 8 bits, ormore, per pixel, and be operable to capture continuous video framesprogressively. Sensor 758 may have a pixel frequency of 170 MHz, ormore. Sensors 758, 759 may include an RGB Bayer color filter, an analogamplifier, an A/D converter, other components to convert incident lightinto a digital signal corresponding to raw image data. Sensors 758, 759may be controlled to operate a rolling shutter or electronic focal planeshutter process where pixels are read out progressively in aline-sequential fashion for a frame. CM 710, 711, and 712 may eachoutput raw data associated with consecutively exposed frames inconformance with any known streaming protocol, such as a MIPI.

In the exemplary embodiment, raw image/video data output by CM 711 and712 is input to ISP 775. ISP 775 is to receive and analyze frames of rawvideo data during the horizontal and/or vertical blanking periodsassociated with CM 710-712. During raw image data processing of RGBimage data, ISP 775 may perform one or more of color space conversion,noise reduction, pixel linearization, and shading compensation, forexample. In some embodiments, raw image data is passed through ISP 775to be processed downstream by a programmable microprocessor 750.

Image data output by ISP 775 may be buffered and queued as input imagedata ready for further image processing, facial detection, and depthdifference determinations in accordance with one or more of theembodiments described elsewhere herein. In embodiments, processor(s) 750includes logic to perform abdominal cavity position tracking andthresholding operations described elsewhere herein. In embodiments,processor(s) 750 includes logic to perform the abdominal cavity movementdistance and rate testing described elsewhere herein. In someembodiments, processor(s) 750 includes logic to perform one or more ofthe operations of method 201 (FIG. 2) and/or method 601 (FIG. 6).

In some embodiments, processor(s) 750 includes face and/or objectdetection logic 751 to detect an infant's facial features, and abdomentracking logic 752 to track changes in 3D spatial coordinates for pointson an abdominal cavity surface over time. In some embodiments, objectdetection logic 751 is implemented with programmable circuitry includingregisters that have been configured through software instruction(s). Insome embodiments, processor(s) 750 includes a 3D coordinate filter tofilter sampled 3D spatial coordinates, for example over a ROI and/orover multiple frames, for example as described elsewhere here. In someembodiments, 3D coordinate filter logic is implemented with programmablecircuitry including registers that have been configured through softwareinstruction(s). In some embodiments, processor(s) 750 includes abdominalcavity movement measurement logic to determine change in cavity surfaceposition over time and to compare that movement to distance and ratethresholds, for example as described elsewhere herein. In someembodiments, abdomen tracking logic 752 is implemented with programmablecircuitry including registers that have been configured through softwareinstruction(s).

Both software and hardware implementations may be well suited toimplementing infant respiratory monitoring in accordance withembodiments described herein. For hardware implementations, face/objectdetection and/or tracking logic 751, as well as abdomen tracking logic752, may be implemented by fixed function logic, for example provided byISP 775. For software implementations, any known programmable processor,including a core of processor(s) 750, an execution unit of a graphicsprocessor, or any vector processor, may be utilized to implement theface/object detection/tracking logic 751, as well as abdomen trackinglogic 752. Processor(s) 750 may be solely responsible for generating arespiratory log from changes in depth values over time determined frominput image data received from ISP 775. In one exemplary embodiment,face/object detection and/or tracking logic 751, as well as abdomentracking logic 752, are invoked through the user space of a softwarestack instantiated by processor(s) 750. In some embodiments,processor(s) 750 executes face/object detection and/or tracking logic751, as well as abdomen tracking logic 752, in a kernel space of thesoftware stack. In some embodiments, processor(s) 750 is programmed withinstructions stored on a computer readable media to cause the processorto perform one or more infant respiratory monitoring operations, forexample as described elsewhere herein.

As further illustrated in FIG. 7, logged respiratory data, and/oralerts, and/or image frames may be output tostorage/display/transmission pipeline 795. In one exemplary storagepipeline embodiment, both respiratory data and one or more issued alertsare written to electronic memory 720 (e.g., DDR, etc.) to supplementimage frame data encoded into a compressed format by A/V encoder 753.Memory 720 may be separate or a part of a main memory 709 accessible toprocessor 750. Alternatively, or in addition,storage/display/transmission pipeline 795 is to transmit respiratorydata, and/or alerts as metadata associated with compressed video data toreceivers remote to camera platform 120.

FIG. 8 block diagrams a data processing system 800 that may be operatedto monitor infant respiration. Data processing system 800 includes oneor more processors 750 and one or more graphics processors 801, and maybe implemented in a single processor desktop system, a multiprocessorworkstation system, or a server system having a large number ofprocessors 850 or processor cores 807. In another embodiment, the dataprocessing system 800 is a system-on-a-chip (SoC) integrated circuit foruse in mobile, handheld, or embedded devices.

An embodiment of data processing system 800 can include, or beincorporated within a server-based gaming platform, a game console,including a game and media console, a mobile gaming console, a handheldgame console, or an online game console. In some embodiments, dataprocessing system 800 is a mobile phone, smart phone, tablet computingdevice or mobile Internet device. Data processing system 500 can alsoinclude, couple with, or be integrated within a wearable device, such asa smart watch wearable device, smart eyewear device, augmented realitydevice, or virtual reality device. In some embodiments, data processingsystem 800 is a television or set top box device having one or moreprocessors 502 and a graphical interface generated by one or moregraphics processors 508.

In some embodiments, the one or more processors 802 each include one ormore processor cores 807 to process instructions which, when executed,perform operations for system and user software. In some embodiments,each of the one or more processor cores 807 is configured to process aspecific instruction set 809. In some embodiments, instruction set 809may facilitate Complex Instruction Set Computing (CISC), ReducedInstruction Set Computing (RISC), or computing via a Very LongInstruction Word (VLIW). Multiple processor cores 807 may each process adifferent instruction set 809, which may include instructions tofacilitate the emulation of other instruction sets. Processor core 807may also include other processing devices, such a Digital SignalProcessor (DSP).

In some embodiments, the processor 802 includes cache memory 804.Depending on the architecture, the processor 802 can have a singleinternal cache or multiple levels of internal cache. In someembodiments, the cache memory is shared among various components of theprocessor 802. In some embodiments, the processor 802 also uses anexternal cache (e.g., a Level-3 (L3) cache or Last Level Cache (LLC))(not shown), which may be shared among processor cores 807 using knowncache coherency techniques. A register file 806 is additionally includedin processor 802 which may include different types of registers forstoring different types of data (e.g., integer registers, floating pointregisters, status registers, and an instruction pointer register). Someregisters may be general-purpose registers, while other registers may bespecific to the design of the processor 802.

In some embodiments, processor 802 is coupled to a processor bus 810 totransmit data signals between processor 802 and other components insystem 800. System 800 has a ‘hub’ system architecture, including amemory controller hub 816 and an input output (I/O) controller hub 830.Memory controller hub 816 facilitates communication between a memorydevice and other components of system 800, while I/O Controller Hub(ICH) 830 provides connections to I/O devices via a local I/O bus.

Memory device 820 can be a dynamic random access memory (DRAM) device, astatic random access memory (SRAM) device, flash memory device, or someother memory device having suitable performance to serve as processmemory. Memory 820 can store data 822 and instructions 821 for use whenprocessor 802 executes a process. Memory controller hub 816 also coupleswith an optional external graphics processor 812, which may communicatewith the one or more graphics processors 808 in processors 802 toperform graphics and media operations.

In some embodiments, ICH 830 enables peripherals to connect to memory820 and processor 802 via a high-speed I/O bus. The I/O peripheralsinclude an audio controller 846, a firmware interface 828, a wirelesstransceiver 826 (e.g., Wi-Fi, Bluetooth), a data storage device 824(e.g., hard disk drive, flash memory, etc.), and a legacy I/O controllerfor coupling legacy (e.g., Personal System 2 (PS/2)) devices to thesystem. One or more Universal Serial Bus (USB) controllers 542 connectinput devices, such as keyboard and mouse 844 combinations. A networkcontroller 834 may also couple to ICH 830. In some embodiments, ahigh-performance network controller (not shown) couples to processor bus810.

FIG. 9 is a diagram of an exemplary ultra-low power system 900incorporating an abdomen distance monitor 930, in accordance with one ormore embodiment. System 900 may be a mobile device although system 900is not limited to this context. System 900 may be incorporated into awearable computing device, laptop computer, tablet, touch pad, handheldcomputer, palmtop computer, cellular telephone, smart device (e.g.,smart phone, smart tablet or mobile television), mobile internet device(MID), messaging device, data communication device, and so forth. System900 may also be an infrastructure device. For example, system 900 may beincorporated into a large format television, set-top box, securitymonitor, desktop computer, or other home or commercial network device.

System 900 includes a device platform 902 that may implement all or asubset of the various respiratory monitoring methods and any of thelogic blocks/circuitry described above in the context of FIG. 1-8. Invarious exemplary embodiments, video processor 915 executes at least oneof pixel depth sampling, filtering, object tracking, and cavity movementmonitoring, for example as described above. Video processor 915 includeslogic circuitry 930 implementing respiratory monitoring based on 3D datafrom an input image, for example as described elsewhere herein.Alternatively, central processor 910 includes logic circuitry 930implementing respiratory monitoring based on 3D data from an inputimage, for example as described elsewhere herein. In some embodiments,one or more computer readable media may store instructions, which whenexecuted by CPU 910 and/or video processor 915, cause the processor(s)to execute one or more 3D image data-based respiratory monitoringmethod, such as any of those described in detail above. A measuredrespiration rate and/or tidal volume determined from the 3D image datamay then be stored in memory 912.

In embodiments, device platform 902 is coupled to a human interfacedevice (HID) 920. Platform 902 may collect raw image data with CM 710,711, 712, which is processed and output to HID 920. A navigationcontroller 950 including one or more navigation features may be used tointeract with, for example, device platform 902 and/or HID 920. Inembodiments, HID 920 may include any television type monitor or displaycoupled to platform 902 via radio 918 and/or network 960. HID 920 mayinclude, for example, a computer display screen, touch screen display,video monitor, television-like device, and/or a television to receivetouch inputs while an input image is displayed on display 922.

In embodiments, device platform 902 may include any combination of CM910-912, chipset 905, processors 910, 915, memory/storage 912,applications 916, and/or radio 918. Chipset 905 may provideintercommunication among processors 910, 915, memory 912, applications916, or radio 918.

One or more of processors 910, 915 may be implemented as one or moreComplex Instruction Set Computer (CISC) or Reduced Instruction SetComputer (RISC) processors; x86 instruction set compatible processors,multi-core, or any other microprocessor or central processing unit(CPU).

Memory 912 may be implemented as a volatile memory device such as, butnot limited to, a Random Access Memory (RAM), Dynamic Random AccessMemory (DRAM), or Static RAM (SRAM). Memory 912 may also be implementedas a non-volatile storage device such as, but not limited to flashmemory, battery backed-up SDRAM (synchronous DRAM), magnetic memory,phase change memory, and the like.

Radio 918 may include one or more radios capable of transmitting andreceiving signals using various suitable wireless communicationstechniques. Such techniques may involve communications across one ormore wireless networks. Example wireless networks include (but are notlimited to) wireless local area networks (WLANs), wireless personal areanetworks (WPANs), wireless metropolitan area network (WMANs), cellularnetworks, and satellite networks. In communicating across such networks,radio 618 may operate in accordance with one or more applicablestandards in any version.

In embodiments, system 900 may be implemented as a wireless system, awired system, or a combination of both. When implemented as a wirelesssystem, system 900 may include components and interfaces suitable forcommunicating over a wireless shared media, such as one or moreantennas, transmitters, receivers, transceivers, amplifiers, filters,control logic, and so forth. An example of wireless shared media mayinclude portions of a wireless spectrum, such as the RF spectrum and soforth. When implemented as a wired system, system 900 may includecomponents and interfaces suitable for communicating over wiredcommunications media, such as input/output (I/O) adapters, physicalconnectors to connect the I/O adapter with a corresponding wiredcommunications medium, a network interface card (MC), disc controller,video controller, audio controller, and the like. Examples of wiredcommunications media may include a wire, cable, metal leads, printedcircuit board (PCB), backplane, switch fabric, semiconductor material,twisted-pair wire, co-axial cable, fiber optics, and so forth.

The thresholded pixel value matching and associated object processescomporting with exemplary embodiments described herein may beimplemented in various hardware architectures, cell designs, or “IPcores.”

While certain features set forth herein have been described withreference to embodiments, this description is not intended to beconstrued in a limiting sense. Hence, various modifications of theimplementations described herein, as well as other implementations,which are apparent to persons skilled in the art to which the presentdisclosure pertains are deemed to be within the spirit and scope of thepresent disclosure.

The following paragraphs briefly describe some exemplary embodiments:

In one or more first embodiment, a computer implemented method forrespiratory monitoring includes processing a sequence of image framesfrom camera image sensor data collected over a time interval,determining a depth map associated with each of the frames, determiningchanges in the depth map over the frames, determining a respiratorycycle over the time interval based on the depth map changes, andgenerating an alert if the frequency or amplitude of the respiratorycycle fails to satisfy one or more predetermined criteria.

In furtherance of the first embodiments, the method further comprisesdetecting and tracking an object within the frames. Determining changesin the depth map further comprises selecting a region of interest (ROI)within the frames based on a position of the tracked object within theframes, and tracking depth values limited to within the ROI over theframes.

In furtherance of the first embodiments immediately above, detecting andtracking the object further comprises detecting and tracking one or morefacial features.

In furtherance of the first embodiments immediately above, the methodfurther comprises generating an alert in response to failing to detectthe one or more facial features.

In furtherance of the first embodiments, determining the respiratorycycle over the time interval based on the depth map changes furthercomprises at least one of spatially filtering depth values for each ofthe frames, temporally filtering depth values over a plurality offrames, and storing the filtered depth values in association with atimestamp.

In furtherance of the first embodiments immediately above, determiningthe respiratory cycle over the time interval based on the depth mapchanges further comprises determining transitions between exhalation andinhalation based on a maximum and minimum of the filtered depth valuesover time, determining a frequency of the respiratory cycle based on thetransitions between exhalation and inhalation, and determining amagnitude of the respiratory cycle based on a difference betweenconsecutive maximum and minimum depth values.

In furtherance of the first embodiments immediately above, analyzing therate or amplitude of the changes comprises comparing the respiratoryfrequency against a predetermined minimum respiration rate threshold,and comparing the respiratory magnitude against a predetermined minimumrespiration tidal volume threshold.

In furtherance of the first embodiments immediately above, generatingthe alert further comprises triggering a first alarm locally, orremotely over a communication network, in response to the respiratoryfrequency failing to satisfy the predetermined minimum respiration ratethreshold for a predetermined period of time, and triggering a secondalarm locally, or remotely over a communication network, in response tothe respiratory depth failing to satisfy the predetermined minimum tidalvolume threshold for a predetermined period of time.

In furtherance of the first embodiments, the method further comprisescollecting the image frames with at least one of a plurality of digitalcameras, a time of flight digital camera, or a structured lightilluminator.

In furtherance of the first embodiments immediately above, the methodfurther comprises illuminating, with a near infrared (NIR) source, atleast a portion of the field of view associated with the camera imagesensor.

In one or more second embodiments, a computerized respiratory monitoringdevice, comprises an input port to receive a sequence of image framesfrom camera image sensor data collected over a time interval, and one ormore processors coupled to the input port. The processors are to processa sequence of image frames from camera image sensor data collected overa time interval, determine a depth map associated with each of theframes, determine changes in the depth map over the frames, determine arespiratory cycle over the time interval based on the depth map changes,store an indication of the respiratory cycle to an electronic memory,and generate an alert if the frequency or amplitude of the respiratorycycle fails to satisfy one or more predetermined criteria.

In furtherance of the second embodiments, a processor to determine therespiratory cycle is to determine transitions between exhalation andinhalation based on a maximum and minimum of the filtered depth valuesover time, determine a frequency of the respiratory cycle based on thetransitions between exhalation and inhalation, NS determine a magnitudeof the respiratory cycle based on a difference between consecutivemaximum and minimum depth values.

In furtherance of the second embodiments immediately above, a processorto analyze the rate or amplitude of the changes is to compare therespiratory frequency against a predetermined minimum respiration ratethreshold, and compare the respiratory magnitude against a predeterminedminimum respiration tidal volume threshold.

In furtherance of the second embodiments, a processor to determine therespiratory cycle is to spatially filter depth values for each of theframes, or temporally filter depth values over a plurality of frames,and store the filtered depth values in association with a timestamp.

In furtherance of the second embodiments, a processor to determinechanges in the depth map is to select a region of interest (ROI) withinthe frames based on a position of the tracked object within the frames,and track depth values limited to within the ROI over the frames.

In furtherance of the second embodiments, the device further comprisesat least one of a plurality of digital cameras, a time of flight digitalcamera, or a structured light illuminator to generate the camera imagesensor data.

In one or more third embodiments, a mobile comprises a mount, a supportarm attached to the mount, the support arm configured to suspending oneor more objects, and the computerized respiratory monitoring devicerecited in any one of the second embodiments.

In one or more fourth embodiment, one or more computer-readable storagemedia includes instructions stored thereon, which when executed by aprocessor, cause the processor to perform a method comprising processinga sequence of image frames from camera image sensor data collected overa time interval, determining a depth map associated with each of theframes, determining changes in the depth map over the frames,determining a respiratory cycle over the time interval based on thedepth map changes, and generating an alert if the frequency or amplitudeof the respiratory cycle fails to satisfy one or more predeterminedcriteria.

In furtherance of the fourth embodiments, the media further comprisesinstructions stored thereon, which when executed by the processor,further cause the processor to perform the method further comprisingdetecting and tracking an object within the frames, and determiningchanges in the depth map further comprises selecting a region ofinterest (ROI) within the frames based on a position of the trackedobject within the frames, and tracking depth values limited to withinthe ROI over the frames.

In furtherance of the fourth embodiments, detecting and tracking theobject further comprises detecting and tracking one or more facialfeatures.

In furtherance of the fourth embodiments immediately above, the mediafurther comprises instructions stored thereon, which when executed bythe processor, further cause the processor to perform the method furthercomprising generating an alert in response to failing to detect the oneor more facial features.

In furtherance of the fourth embodiments, determining the respiratorycycle over the time interval based on the depth map changes furthercomprises at least one of spatially filtering depth values for each ofthe frames, temporally filtering depth values over a plurality offrames, and storing the filtered depth values in association with atimestamp.

In furtherance of the fourth embodiments, the media further comprisesinstructions stored thereon, which when executed by the processor,further cause the processor to perform the method further comprisingspatially filtering depth values for each of the frames, temporallyfiltering depth values over a plurality of frames, storing the filtereddepth values in association with a timestamp, determining transitionsbetween exhalation and inhalation based on a maximum and minimum of thefiltered depth values over time, determining a frequency of therespiratory cycle based on the transitions between exhalation andinhalation, and determining a magnitude of the respiratory cycle basedon a difference between consecutive maximum and minimum depth values.

In one or more fifth embodiments, a computerized imaging device,comprises a means to perform any one of the first embodiments.

In one or more sixth embodiments, computer-readable storage mediaincludes instructions stored thereon, which when executed by aprocessor, cause the processor to perform any one of the firstembodiments.

In one or more seventh embodiment, a computerized respiratory monitoringdevice, comprises a means to receive a sequence of image frames fromcamera image sensor data collected over a time interval, a means toprocess a sequence of image frames from camera image sensor datacollected over a time interval, a means to determine a depth mapassociated with each of the frames, a means determine changes in thedepth map over the frames, a means to determine a respiratory cycle overthe time interval based on the depth map changes, a means to store anindication of the respiratory cycle to an electronic memory, and a meansto generate an alert if the frequency or amplitude of the respiratorycycle fails to satisfy one or more predetermined criteria.

In furtherance of the seventh embodiment, the device further comprises ameans to generate the camera image sensor data.

It will be recognized that the embodiments are not limited to theexemplary embodiments so described, but can be practiced withmodification and alteration without departing from the scope of theappended claims. For example, the above embodiments may include specificcombination of features. However, the above embodiments are not limitedin this regard and, in embodiments, the above embodiments may includeundertaking only a subset of such features, undertaking a differentorder of such features, undertaking a different combination of suchfeatures, and/or undertaking additional features than those featuresexplicitly listed. Scope should, therefore, be determined with referenceto the appended claims, along with the full scope of equivalents towhich such claims are entitled.

What is claimed is:
 1. A computerized respiratory monitoring device,comprising: an input port to receive a sequence of image frames fromcamera image sensor data collected over a time interval; and one or moreprocessors coupled to the input port, the processors to: process asequence of image frames from camera image sensor data collected over atime interval; determine a depth map associated with each of the frames;determine changes in the depth map over the frames; determine arespiratory cycle over the time interval based on the depth map changes;store an indication of the respiratory cycle to an electronic memory;and generate an alert if the frequency or amplitude of the respiratorycycle fails to satisfy one or more predetermined criteria.
 2. The deviceof claim 1, wherein a processor to determine the respiratory cycle isto: determine transitions between exhalation and inhalation based on amaximum and minimum of the filtered depth values over time; determine afrequency of the respiratory cycle based on the transitions betweenexhalation and inhalation; and determine a magnitude of the respiratorycycle based on a difference between consecutive maximum and minimumdepth values.
 3. The device of claim 2, wherein the processor is furtherto: compare the respiratory frequency against a predetermined minimumrespiration rate threshold; and compare the respiratory magnitudeagainst a predetermined minimum respiration tidal volume threshold. 4.The device of claim 1, wherein a processor to determine the respiratorycycle is to: spatially filter depth values for each of the frames; ortemporally filter depth values over a plurality of frames; and store thefiltered depth values in association with a timestamp.
 5. The device ofclaim 1, a processor to determine changes in the depth map is to: selecta region of interest (ROI) within the frames based on a position of thetracked object within the frames; and track depth values limited towithin the ROI over the frames.
 6. The device of claim 1, furthercomprising: at least one of a plurality of digital cameras, a time offlight digital camera, or a structured light illuminator to generate thecamera image sensor data.
 7. A mobile, comprising: a mount; a supportarm attached to the mount, the support arm configured to suspending oneor more objects; and the computerized respiratory monitoring devicerecited in claim
 1. 8. One or more computer-readable storage media, withinstructions stored thereon, which when executed by a processor, causethe processor to perform a method comprising: processing a sequence ofimage frames from camera image sensor data collected over a timeinterval; determining a depth map associated with each of the frames;determining changes in the depth map over the frames; determining arespiratory cycle over the time interval based on the depth map changes;and generating an alert if the frequency or amplitude of the respiratorycycle fails to satisfy one or more predetermined criteria.
 9. The mediaof claim 8, further comprising instructions stored thereon, which whenexecuted by the processor, further cause the processor to perform themethod further comprising: detecting and tracking an object within theframes; and wherein determining changes in the depth map furthercomprises: selecting a region of interest (ROI) within the frames basedon a position of the tracked object within the frames; and trackingdepth values limited to within the ROI over the frames.
 10. The media ofclaim 9, wherein detecting and tracking the object further comprisesdetecting and tracking one or more facial features.
 11. The media ofclaim 10, further comprising instructions stored thereon, which whenexecuted by the processor, further cause the processor to perform themethod further comprising: generating an alert in response to failing todetect the one or more facial features.
 12. The media of claim 8,wherein determining the respiratory cycle over the time interval basedon the depth map changes further comprises at least one of: spatiallyfiltering depth values for each of the frames; temporally filteringdepth values over a plurality of frames; and storing the filtered depthvalues in association with a timestamp.
 13. The media of claim 8,further comprising instructions stored thereon, which when executed bythe processor, further cause the processor to perform the method furthercomprising: spatially filtering depth values for each of the frames;temporally filtering depth values over a plurality of frames; storingthe filtered depth values in association with a timestamp; determiningtransitions between exhalation and inhalation based on a maximum andminimum of the filtered depth values over time; determining a frequencyof the respiratory cycle based on the transitions between exhalation andinhalation; and determining a magnitude of the respiratory cycle basedon a difference between consecutive maximum and minimum depth values.14. A computer implemented method for respiratory monitoring,comprising: processing a sequence of image frames from camera imagesensor data collected over a time interval; determining a depth mapassociated with each of the frames; determining changes in the depth mapover the frames; determining a respiratory cycle over the time intervalbased on the depth map changes; and generating an alert if the frequencyor amplitude of the respiratory cycle fails to satisfy one or morepredetermined criteria.
 15. The method of claim 14, further comprising:detecting and tracking an object within the frames; and whereindetermining changes in the depth map further comprises: selecting aregion of interest (ROI) within the frames based on a position of thetracked object within the frames; and tracking depth values limited towithin the ROI over the frames.
 16. The method of claim 15, whereindetecting and tracking the object further comprises detecting andtracking one or more facial features.
 17. The method of claim 16,wherein the method further comprises generating an alert in response tofailing to detect the one or more facial features.
 18. The method ofclaim 14, wherein determining the respiratory cycle over the timeinterval based on the depth map changes further comprises at least oneof: spatially filtering depth values for each of the frames; temporallyfiltering depth values over a plurality of frames; and storing thefiltered depth values in association with a timestamp.
 19. The method ofclaim 14, wherein determining the respiratory cycle over the timeinterval based on the depth map changes further comprises: determiningtransitions between exhalation and inhalation based on a maximum andminimum of the filtered depth values over time; determining a frequencyof the respiratory cycle based on the transitions between exhalation andinhalation; and determining a magnitude of the respiratory cycle basedon a difference between consecutive maximum and minimum depth values.20. The method of claim 18, further comprising: comparing therespiratory frequency against a predetermined minimum respiration ratethreshold; and comparing the respiratory magnitude against apredetermined minimum respiration tidal volume threshold.
 21. The methodof claim 19, wherein generating the alert further comprises: triggeringa first alarm locally, or remotely over a communication network, inresponse to the respiratory frequency failing to satisfy thepredetermined minimum respiration rate threshold for a predeterminedperiod of time; and triggering a second alarm locally, or remotely overa communication network, in response to the respiratory depth failing tosatisfy the predetermined minimum tidal volume threshold for apredetermined period of time.
 22. The method of claim 14, furthercomprising: collecting the image frames with at least one of: aplurality of digital cameras; a time of flight digital camera; or astructured light illuminator.
 23. The method of claim 22, furthercomprising illuminating, with a near infrared (NIR) source, at least aportion of the field of view associated with the camera image sensor.